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***Title of the Invention*** 

THREE-DIMENSIONAL SHAPE MEASURING METHOD AND ITS DEVICE 
**Summary of the Invention** 

A three-dimensional shape is measured by a simple system structure. A 
three-dimensional shape measuring instrument comprises a device (l"l) in which a 
light- emitting diode (l'lb) is installed as a marker in a line laser light source (l-la), an 
imaging device (1-2), and a computer (1*3). For measurement, a line laser beam from 
the device (l-l) is applied to an object (1*4) to be measured, the imaging device (1-2) 
images the applied line laser beam (1-5) and the Ught-emitting diode, and a 
three-dimensional shape is obtained from the image data by triangulation by means of 
the computer (1-3). 

***** Description ******* 

**Field of the Invention** 

This invention is related to 3D measurement system which use laser-light and estimate 
depth information of an object by applying triangulation method; among these systems, 
this invention is especially related to 3D measurement system which consists of a single 
camera to acquire accurate depth information. 

**Background of the Invention** 

To acquire 3D shape of the object, active stereo method based on triangulation method is 
well known (pattern projection method and point projection method). These systems 
consist of light sources such as laser-lights or halogen lights and image capturing 
devices such as cameras or CCD(Charge-coupled Device) cameras. 3D shape estimation 
is conducted by triangulation method of these devices. Therefore, precise relationship 
between the light source and the camera should be precalibrated, and thus, the whole 
system is prone to become large and complicated. 

If we can make calibration process of the light source and the camera easier, active 3D 
acquisition system becomes simpler. Based on such idea, the method which achieves 



easy calibration of the light source and the camera by attaching markers on the light 
source itself and capturing it with a single camera was proposed in (Masahiro 
Takatsuka, Geoff A.W. West, Svetha Venkatesh,and Terry M. Caelli. "Low-cost 
interactive active monocular range finder". In Proceeding of Computer Vision and 
Pattern Recognition, 20 volume 1, pages 444-449, 1999.). However, since their method 
assumes a beam of light for the light source, and thus, 3D information of only a single 
point can be acquired from a single image, it causes a serious problem that it takes a 
very long time to capture 3D data of a large area. Another problem of their method is 
that, since markers attached to the light source are aligned along a single line, precision 
of acquired 3D information is inevitably low. 

Thus, our invention aims to reduce the capturing time and improve precision of 3D data 
of 3D measurement system based on active stereo method in which a marker attached 
light source and a single camera are used. 

**Disclosure of the Invention** 

This invention achieves dense 3D data acquisition from a single image by using a line 
laser as the light source. With this feature, 3D shape can be efficiently acquired in a 
short time. In addition, because of the three-dimensional configuration of markers, the 
light source position can be estimated accurately, and thus, precision of 3D data can be 
improved. 

Since proposed method can acquire dense 3D points from a single image, if the acquired 
3D points include 3D points already estimated in past procedure, position of fight source 
can be corrected by using the 3D points which are already estimated, and thus, accuracy 
of 3D shape can be improved. 

**Brief Description of the Drawings** 

The figure 1 shows a configuration of the measuring devices of image data as the first 
embodiment of this invention. The figure 2 shows a configuration of the measuring 
devices of image data as the second embodiment of this invention. The figure 3 shows an 
example of how to arrange the Ught-emitting diode(LED) markers, which is used in this 
invention. The figure 4 shows the relationship between the marker coordinate system 
which are fixed to the line-laser source and the camera coordinate system. The figure 5 



shows the method for estimating the rotation of the marker coordinate system. The 
figure 6 shows the method for estimating the positions of the 3D points used in this 
invention. The figure 7 shows the method for estimating the difference between the 
marker plane and the laser plane. 



**Description of the Preferred Embodiments** 

In the following, an implementation of this invention is described, referring the figures. 
First, we disclose the implementation of 3D measurement process using figure 1. At 
first step, the users set up all objects in order to image capturing device(l"2) can capture 
both light projecting device(l'l) and target object(l-4) in a same frame. At the same 
time, an iris value and shutter speed are adjusted so that the laser light and LEDs are 
accurately and simply detected by cameras by using image processing method. In 
addition, the relationship between light projecting device (l-l) and the sheet of light 
formed by line laser are measured in advance. In terms of image capturing device, we 
can use either video camera, CCD camera and so on. At the second step, the users hold 
the light projecting device(l-l) by hand and project the laser onto the measured object. 
Since 3D configuration of LEDs are known, by detecting the positions of LEDs on image 
plane, the relationship between light projecting device (l-l) and camera can be 
calculated. For calculation, nonlinear optimization are conducted to improve accuracy. 

Since relationship between light projecting device (l-l) and laser plane is known, we 
can recover depth of light projected area from camera with triangulation method using 
the relationship between the light projecting device (l*l) and the camera. 3D shapes of 
the target objects are obtained by repeating the process. If obtained 3D shapes are 
displayed on a display device (2*5) in real-time as shown in Fig.2, the users can learn 
that the area where does not measured yet on site. Thus, the users can efficiently scan 
the 3D shape of the target object. 

Examples of marker configuration are shown in Figure 3. If number of marker is more 
than 4, marker configuration is basically free; however, square shaped configuration 
(3-1), xyz coordinate shaped configuration (3-2) and cube shaped configuration (3-3) 
have some advantage such as easy manufacture and simple calculation for retrieving 
the relationship between light projecting device (l _ l) and camera. 



In the following, concrete implementation of 3D measuring device and the method are 
described. 

Our system consists of a video camera and a laser-projecting device (See Fig.l). While 
measuring, the users hold the laser projector by hand and project the laser onto the 
measured object. The projected stripe is observed by the video camera. The shape of the 
target object is obtained by analyzing the video sequence. To estimate the position and 
orientation of the line laser-projecting device, LED markers are attached to the laser 
projector. LED markers are located so that they form a square and so that the square is 
placed on the same plane as the sheet of light. 

These markers define the local coordinate system in 3D space (4-1). We call this the 
"marker coordinate system" in this description. The origin of the marker coordinate 
system is placed at the center of the markers and we define the marker coordinate 
system by three unit vectors, el', e2' and e3'. The first and the second vectors, whose 
directions are expressed by el' and e2* are taken so that they are parallel to the edges of 
the square formed by markers, (el' is parallel to the light projecting direction of laser 
projector and e2' is perpendicular to the direction.) The direction of the third vector e3' 
is orthogonal to the plane formed by the four markers. We also define the camera 
coordinate system by el, e2 and e3, where el and e2 are respectively parallel to 
horizontal and vertical directions of image plane, and e3 directed towards the backward 
direction of the camera (4-2). The center of the camera coordinate system is defined as 
the center of the lens. See Fig.4. 

Here, we name 2 planes. The first one is the plane spanned by el' and e2', which we call 
the "marker plane. "(5-4) The second one is the plane where the laser light is projected, 
which we call the "laser plane." The transformation between "marker plane" and 
"laser plane" are represented by a 3D rotation matrix R and translation vector t. This 
transformation represents the marker positions from the camera. See Fig. 5. When the 
shape of the target object is scanned, the emitted light from markers are captured by 
video camera. From the captured image frame, the locations of the four LEDs are 
obtained. In actual implementation, it can be achieved by simple thresholding and 
calculating the center of gravity of the connected pixels. By using the locations of LED 
markers, a description of the lines on the image which go through the markers is 
calculated and the crossing point of the lines are also calculated. The crossing points are 



called a Focus of Expansion (FOE).(5"2a) and (5~2b). If the lines are parallel on the 
image, the FOE is defined as the point at infinity at the direction of the parallel lines. 
Let the 2D coordinates of these vanishing points be described as {g 0n ,gi n ){n = 0,l) in 
pixels, then, the 3D direction vectors (rl and r2) of those FOEs are 

r n ={gon>gxn-f) 

where f denotes focal length in pixel and °' denotes transpose of vectors and matrices. 
We define e\ = r,/||r f ||,(/ = 1,2) . The unit vectors e\ are used as estimation of e\ . 

Estimation e\ is defined by taking a cross product of the two unit vectors as 
e\ =e[xe' 2 . Figure 5 shows relationships between FOE and estimations e' i9 (i = 1,2). 

Using three unit vectors, estimated R of rotation matrix R can be described as- 

R = (e l9 e 29 e 3 ). (2) 

Let 2D coordinates of images of first and second markers be {p u > Pit) P22) - 3D 
locations in camera coordinates of the markers are 

m \ = (Pu u >Pi2 u -f u y> m 2 = (^2i v ^22 v -f»y (3) 

where w,v are unknown variables. Let the 3D distance between those markers, which 
is known value, be Dm. Then 

H-m 2 \\ = D m . (4) 

The equation of the marker plane is expressed as e' 3 [ x + d = 0 , where x is a 3D point 
expressed in camera coordinates. Since m 1 ,m 2 are on the marker plane, 

+4 = 0(1 = 1,2). (5) 

From equations (3)(4) and (5), d,u,v can be solved. Then we obtain t , as an estimation 
of t , by calculating the center of gravity of locations of the markers. 

To improve the estimation of the parameters of transformation, we optimize it with a 
nonlinear optimization. The evaluation function for optimization is defined as the sum 
of the squared distances between the actual LED positions on the image and the 
estimated marker positions on the image, which are calculated by projecting the LED 

position in local coordinates to image plane by using estimated R and i . The function 
is described as 



f{R,t)^\proj(Rm' i+ t)- Pi } (6) 

i 

proj^x, ,x 2 ,x 3 )')= (O, )/x 2 , (/x 2 )/* 3 )' (7) 

where p i is a location of image of the / th marker, m\ is a location of the i th 
detected marker in marker coordinates. projO is an operation of projection from 3D 
position in camera coordinates into image coordinates. For actual implementation, we 
can use simplex descending algorithm for the optimization. 

Then, using the estimated parameters of the marker plane, we can estimate the 
parameter of the laser plane. Detailed method will be described later. 
First, the pixels which are projected by laser light (6~l) are extracted from the image 
(6-5). The pixels are easily picked up by simple thresholding method, however, for actual 
implementation, skeletonizing and/or morphological filter are applied for improving 
accuracy. The line going through the origin of the camera coordinates (6-4) and the 
surface point can be calculated for each pixel using intrinsic parameters, and the 3D 
location of the surface(6*3) can be determined by taking the intersection of the line and 
the estimated laser plane (6-2). See Fig.6. 

Here, we explain the method to acquire correct relationship between the marker 
coordinates and the laser plane. The relationship can be defined as the parameters of 
the laser plane expressed in marker coordinates. The parameters can be expressed by (a, 
b, c, d) which represents plane in 3D space. If the markers plane is precisely correspond 
to the laser plane, the equation of the laser plane is z=0. 

For preparation of the calibration between the marker coordinates and the laser plane, 
the following measurement must be conducted in advance. First, a box-shaped object 
(7-5), of a known size is captured. From the image, the extrinsic camera parameters are 
estimated using methods of camera calibration. From the extrinsic camera parameters 
and the size, the equations of the planes enveloping the box are estimated. 

We project the laser light onto the surface of the target object, and capture the lit 
surface and the markers. From the markers, the transformation between the marker 
coordinates and camera coordinates can be estimated by previously described method. 



Then, the laser plane expressed in the camera coordinates can be calculated from the 
estimated laser plane parameters (a, b, c, d) expressed in marker coordinates. Next, the 
intersecting lines of the laser plane and the surface of the box shaped object are 
calculated. Then, the images of the lines can be determined (7-2). 

We calculate the intersection of the laser plane estimated by the above method and the 
box-shaped object. Then, we estimate the image of the intersection and compare the 
image with the true positions of the image of the surface points lit by the laser (7-1). The 
squared distances between the pixels of the true image and the estimated image are 
summed up for all the pixels. The sum is the estimation function of the parameters (a, b, 
c, d) which depicts the laser plane expressed in the marker coordinates. By minimizing 
the estimation function using a non- linear optimization method, we can estimate (a, b, c, 
d). The initial value for the optimization can be, for example, z=0, which is equivalent 
with (a,b,c,d)=(0,0,l,0). See figure 7. 

Finally, we describe a method to correct the estimation of the marker coordinate 
system from known 3D points. When the 3D locations of some part of the measured 
surface are known and if some part of the projected stripe of laser is on the region of 
surface whose depth values are known, the known depth value and the depth value 
calculated from the estimated laser plane should match. Suppose we define an 
estimation function which is the sum of function (6) and the sum of squared differences 
between the known depth value and the calculated depth value expressed above. The 
summation of the squared differences is calculated for all the region of surface whose 
depth values are known. Then, we can correct the estimation of the marker coordinate 
system by minimizing the estimation function. As for the known 3D points, we can use 
the 3D points precisely measured by other methods. Another example for the known 3D 
points is that we can select points which are measured many times and the variances of 
whose positions are small. 

**Industrial usability** 

As explained above, applying this invention, we can quickly acquire precise 3D shape 
without using complex equipment which is conventionally required for active 3D 
measurement by triangulation. Moreover, the users can interactively check the regions 
whose positions are not measured while the measuring process by displaying the 
acquired 3D positions in real-time, thus, measurement of 3D shape of the target object 
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7-1 Detected laser light by image processing 

7-2 Intersection of the estimated laser plane and box 

7 3 Captured image 7-4 Video camera 7-5 Box-shaped object of a known size 



